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ABSTRACT 

Described is the Empire State Office of Besearch and 
Evaluation multiple perspective strategy for evaluation of academic 
programs. The uncommon Studio Arts Program in New York City is used 
as a model. This program is politically as well as e^ducationally 
important to the institution, which introduces internal and external 
pressures on the research. These are discussed along wifh the design, 
which • included a variety of qualitative and quantitative methods and 
the policy decision outcomes ©f the evaluation. (Author) 
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^.valuation of academic programs- -theii^ processes, orientations, objectiveSo . 
ptoducts, and impact-- is a recent and important dimension of institutional ' 
research.: Given the increasirig scrutiny under which postsecondary institutidus 
((Operate, educational evaluation is an almost natural outgrowth of the descriptive 
studies and cost analyses that have characterized the field. However, much 
evaluation tdf date relies upon single or at best dual perspectives on effective- 
ness. Given the growing recognition of Various interests both internal and external 
to postsecondary education, this traditional strategy for evaluatiqn is insuf- 
ficient. Needed are designs that provide a rich variety. of data in respjonse to the . 
multiple perspectives and pressures (Carapbe^ll and FisJcQ,' 1959; Webb et al , 1965; 
Sieber, 1973.) This paper outli^s such a design by describing briefly 
the Program Effectiveness and Related Cost (PERC) framework developed at 
Empire State College in conjunction wit;h some other instituti*ons . The paper then 
illustrates how the- multiple perspectives evaluation strategy proved effective in 
ari examination of a complex, innovative. Studio Arts Semester in Nqw York City. " 

irogVam Effectiveness and Related Costs (PERC) 

PERC began three years ago as a project to devjalop ways of linking assessment 

of effectiveness to cost analyses. Jointly sponsored by the Fund fot the Inprove- 

ment of Postsecondary Education, Exxon and Empire State College, PER^ now involves 

p . It 
four^ cooperating institutions: Hampshire College, Amherst, Mass.; Northland 

College, Ashland, Wise; State University of New York, at Plattsburgh, and University^ 
of Wisconsin-Green 'Bay. The framework looks seriously at relationships between out- 
comes and costs for students, faculty* and programs in"an attempt* to leani "what \ 
. kinds of students chang^/ in what, ways following wh^t kinds of educational 'experiences 
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and at what costs?*' Fhe PERC framework has several important features (discussed 
in Pa>ola et al; 1975; Palola ,et al, 1976) but the one that is most pertinent for ' 
his paper is the mult'iple perspectives strategy. 

. , ' i' 

The multiple perspectives strategy developed for the PERC framework is shown 
in Figure 1. It contains three key ideas: first, student learning is the focus 
of PERC evaluations, and thus its placement at the center of the figure; second, 
multiple persons make bbservations of student learning utilizing various ways of 
knowing antl ways of evaluating. This rich composite of diverse information is 
critical to th6 pluralistic notion of evaluation presented here; and third, an 
iTidependent or neutral research an.d evaluation staff synthesizes all data^pes 
and prepares useful reports fpr different audiences. Reports tailor-made to audi- 
ence interests maximizes communication and program improvement based on research - 
data, PERC's aim is clearly to enhance executive decision-making capabilities. 

A few more details. An, important concern of PERC is tjo utilize a variety of 

data collection methods . Student learning ^and growth is a complex process, not yet 

'' 1 ' V , ■ ^ • 

well understood, so reliance upon a single method of data collection is risky. For 
example, test scores provide, according to many, a reasonable estimate of cognitive 
achievement. But, at best, tests look at student' mastery of content. Better also 

to find other methods of data colleption. Interviews, rating forms, survey instru- 
' merits', content ai^alysis, observatidn^d tests are all important tools for obtaining 

a full picture of what is happening, to students. Note that this means use of quali- 

tative as well as quantitative techniques. The aim is to develop chains of evidence 

• • "■ ' " , 

of where program impacts have occurred. ^ ^ 

/" 

Multiple observers and multiple standards are key compt)nents to a multiple 
perspectives strategy. While faculty traditionally assess, student learning and ' 
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program effect iveness, t>students and others including educational researchers also 
can provide richnes^s to an evalu|'tion. Since effectiveness is often in the eyes 
of the beholder, ;iliultiple observ§|5c*s applyang their own standards are important 
to informed decision-making. '(^'* | - . ^ ' 

Overall, the multiple persp^pives approach posits certain common evalua-^ 



tions 'that will be reflected in thjf^ various research techniques as well as certain 
rinuiuc judgments, that may rBflec^; the particular \^ntage*-pQi"ts and knowledge of a 
i^ivt n eval^ator. This approach Conceptualizes^ t]^:e learning/jteaching process as 
cample^v^^lnteract i ve, and unique but also patt'eiWed. As a result, for certain kinds 
ut students working with certain kinds of faciylty, there ought to be identifiable 
.ommon out^^omes. llie multiple perspective /Approach should be able to reveal them. 
Hie researchers task is then to present them in understandable and convincing ways 
r*' the variety of audiences . 

\n Illustration: Hvaluating the Studio Arts- Semester in New York 

The tmpire State College Studio Arts Semester (SAS) in New York was created 

In 1975 to provide an ^opportunity for college undergraduates from throu|hoat the 

State University of New York to spend a semester in the; arts capital off'North 

America. The program is designed for serious, professionally-oriented students 

and allows them to immerse themselves in their media. 

^ The elements iDf the program are simple. Students live in the City and 

) . 

work in a large loft space- connected with the Westbeth^complex on the edge of 
Greenwich J/^llage. All have a personal studio area in the loft where they 
pursue their art interests. In additioif, there are several seminars in courses 
at such City , programs as the Chambers Street Sculptors Workshop, Westbeth Graphic 
Workshop, and the Baldwin Pottery Workshop. Others do ^apprentice work with prac- 
ticing artists. However, the most important |limension of the SAS is regularly 



scheduled critiques in which a professional aivt^iis^/crit ic meets individually with 
each student and offgrs encouragement, suggestions, help, or alternatives. The 
sessions are often intense and test the confidence and mettle of every student. 
Each artist/critic (a total of six) makes 3-4 visits per semester. 

Empire State College>has taken program evaluation -seriously since its 
founding in 1971. - PERC "lool^s at the entire College. Thus, it was not unusual 
when the Office of Research and Evaluation was asked to conduct a study on SAS ^ 
during its initial terra. However, there were several pressures under which the 
staff members worked. ^ \ • ^ 

Decisions were pending about SAS personnel. It was thus necessary to " 
clarify ORE'S rc^lfeV^ The position taken by ORE was that personnel decisions were 
not part of its v/orki. Thus the evaluation stuck to student learning outcojnes and 
the actual operation of the Studio Arts Program^. A second pressure arose from the 
Itiple funding base fqlr the program. Each contribytor held certain interests and 
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these were crucial tq t^m as well as to the success of the Studio Arts Program. 
Thitd, relationships between the students home campus' and SAS wds key. Some art 
department chairmen played a major role in determining which students participated. 
Students often asked about, or openly criticized, admissioifs criteria and the general, 
level of development of students admitted tg^-the painting program. These were some of 
the pressures playing on SAS and ORE at the time of the .evaluation study. 

Six methods of data collection were used. The most basic was gleaning avail- 
able files on enrolled students to discover such, things as age,, sex, and home 
institution. This task might have been easier if files had existed but the combina- 
tion of a new program and uncleax administrative procedures led to a dearth of 
accurate information, • 

The second technique was personal interviews with scudents at the V/estbeth 
loft. The interviews sought to learn why stud-ents enrollq^d, whether the program fit 



neatly into their long-term plans, '^low they v^ere *»using" the City, wh^at the SAS 

provides that is'^riot available on campus, perceived benefits, and problems with 

• ■ *. ^ * 

the program. All stu4ents were interviewed. ^ ' . 

A third technique Has interviews with the Program Coordinator who is a 

\^ 

practicing New York artist and three artist/critics. Thele conversations focused 
especially on SAS j>trengths and weaknesses. 

'A fourth aspect of the methodology was phone interviews with Advisory Board 
members. The Advisory Board consists of art faculty throughout SUNY, ESC - 
administrators, and personnel representing SUNY's Central staff, who helped create 
the SAS and are responsible for its progress. . 

The f4jfth technique was a short anon^ous questionnaire which- utilized open- 
ended quostion:^ about student goals, problems, le^ings, and the like. Mos.t students 
(9(.r6) completed the form. Also, students kept loWs of their work activities and 
react u)Ms tu the program. \ 

Sixth, the research team spent four da^s and two evenii^gs in non-participant \ 
ob>'ervation of stu^eats working, interacting, and undergoing critique. W^ile 
inv'isibiiity is impossible, it seems fair to suggest that w^ Wre not prominent 
j.i the Students* minds. were simply visitors to ignore or talk with depending • 
on time, moodj'^or circumstance. ' ' . * ' 

An^tlv^r^pect of the study was a cost analysis. This compared SA^ total 
and pe^student cos^ts with all-Coiaege figures. 

Overall, the multiple methods and sources provided a ri(^ melange Df inf oi^ . 
mat ion, about the Empire Studio Arts Semester. This was put in as initial report 
on July 30, 1975 for the Advisory Bpard of the University Wide Council on the Arts 
and the Empire administration. The report contained eleven recommendations . ^ In 
fall 1975, the study was replicated, in shorter form in keeping with the PERC re- 
quirement of using longitudinal data whenever possible, ITiis led to a Second report 
in March 1976 which reviewed the disposition o^ the recommendation,. 

■ ... 8 ■ . ' X 
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The Studio Arts Semester ^Report : Findings . 

When the Office of Research and Evaluation began its 
possible that we were merely sliding the ^stone over a grave. The^e 

fusion in and outside of the College about^the SAS. We heard ab\)ut students 3^^^ 

floundering about and. accpjnplishing little. Also, rumors of exorbitant cost were 

rife. We foimd some problems but the following was the major conclusion: . 

Overall, we found the program stimulating and promising. The concept 
is exciting! The studehts are enthusiastic and serious. The 'faculty 
is committed. What prqbl ems that do '^Xist are repais^ble. Thus, we 
enthusiastically recommenS continuati^t^of the Studio Arts Semester 
in New York City. (Palola ^nd Bradley,' 75, Introduction); 

However, there were other findings. 

A^key dimension of the SAS is the opportunity for top quality student artists 

to work closely and interact with other top students. Thus, student selection is 

iriportant. Wl^ife'Ce found. all studenirs enthusiastic and serious, several people 

expressed the cbncern that quality was uneven. In addition, our ^nversations with 

students suggested that many had taken a lot of ^ timft getting comfortable in and 



^ tb^refore able to use the City. Thus, we recommend a strict screening process for 
\_ students that included review of a portfolio. We al^so reSommended a systematic. 



but not overly directive orientation program. 

Problems also appealed in the critiqufes. Students found that the artist/critics 
tended to be highly subjective. )\s one sti^ent noted: "If your work is similar to 
the critic, he loves you.^ But if jiot, forget it!" While such subjectivity is ^ ; 
given in art, the apparent resulting disparity in amount of time spent with students 
is nof a given in education. The disparities were heightened by the amount of work 
for an 'artist/critic in a given day. Thus, we recommended more visits and clearer 
distribution of thei\ wprklfea^ - M , 
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Another area of reconunendation related to the SAS students, who were, 

save one', all primarily painters. Several, however, wanted introduction to ' 

'...If 

other media and tool«, courses and worksjiops in them. To stimulate opportunities l j 

/ ■ - ■ " . 

learning in other meaia, we recomm^ed that students with other than painting 
interests be recruited. ^ ^ ^ 

Facilities of the SAS are three: the loft, the galleries and museums, and 
living accommodations. The loft is a 5000 square- ;^^pt ii^fea which allows 100-150 
square feet per student. Temporary partitions of plaster board separal^e the students 
work spaces. While students seldom complained and even liked the loft, the critics 
did, noting: "New York is dingy endugh. The students should have some way to get 
away from i-t." Another condemned -the place as dirty, poorly lit, located in a high 
risk area, and generally <^sui table. t 

The seco^ elenj^nt' of facilities, galleries and museums are, of course, 1^ 
comparable. Students have Teasy access to a huge variety of learning resources. 
However, in the initial Studio Arts Semester, living accommodations were a serious, 
problem. A few students found apartments, a few lived in the all-female Christian 
Union Building, but the six who chose the' Hotel Albert - a cockroach infested. 



s'kidrow home for all manner of rogues - have the most to remember. The Hotel Albert 
is now immortalized in the loft graffiti for succeeding generations of students. 
Our recommendation under the facilities heading called for some tidying up of the ^ 
loft, purchase of a few useful items -- electric saw for frames, a ladder, some 
more desks and chairs, a slop sink - and a concerted effort identify suitable ^ 
housing. 

The cost analysis, revealed a program that was somewhat more expensive. than ^ 
. the normal Empire State College program. However. . planned future enrollments would 
reverse this picture, (kir recommendations were to look into extending J:he SAS, 
Empire's only semester program, throughout the summer since the Director is ,oa year- 
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round appointment and the loft space is rented on an annual basis. We also sug- 
gested an evaluation of the fees going to artist/critics to ensure that the amounts 
were, reasonable in terms of going rate. . . 

A major problem for students afld the Director was the lack of clarity over 
administrative procedures. The Director felt that he needed his budget earlie^^ 
while students were suffering multiple problems in trying to pay tuition and fees 
and initiate transfers in financial aid moniafc Our recommendation here was to 



straighten thiags out fast before these procedural problems destroyed the program, 
A troncern felt* by the research staff was that several S AS students had no ^ 
intention of returning to the home campus. Since the program is dependent upon t|ie 
^ assist'ince and encouragement, of art faculty throughout SUNY/^this seemed a seriou^ 
problem. Thus, our final recommendation was that considerable care must be,. taken 
> , to ensure that home campus art departments realize suitable benefits. y 
IVhile there is nothing extraordinary about the initial Studio Arts Semester 
report (we, in fact, consider it fairly routine in the context of PERC) , i^t provides 
a clear example of the advantages of a multiple perspe9tive5 strategy. Withou^ 
multiple observ ers ^ several problems might never have been identified. Withouti / 
multiple methods, we would have had. great difficulty offering interpretations o| 
■ W • re iative "importance of the various recommendations_^_^iiJiiiarly, multiple eval- 

,i;*rurs [with different standards) of the draft report helped make the final 
v^roduct powerful. Here are some observations from the second report (Ps^^lola 

:r. . Br idlev , 19 76 I . . J ■ , • ' ^ 

i mpact of the Initial" Studio Arts Semester Report. 

Xhe major impact of the' initial report is that the SAS continues with 
generally erithusiast:i|pt: support ftom Empire State College and the^rest of the State 
University. While there are still concerns, all seem now convinced that the'pro- 
gram is serious and productive. ^ - , ^ 

^ ITie fall 1975 SAS was different from the spring in several' ways . First, there 

ERJC 11 . . • ■ 
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were more female students ,^more older students/ and the artist/critics seema^ to. 
feel that the overall quality was higher'. Second, the loft was more crowded) as 
numbers increased. As one researcher found, it was a simple matf^e^to trip on a 



painting or Step on a tube of paint, ThTrd, students acclimared to the City sooner. 
This was reflected in the number who made arrangements for activities outside the 
loft; apprentic-eships , courses, work with tutors, and the like. Fourtli, the 
critiques seemed to work better. - • ^ ' * 

The Office of Research and Evaluation recommendations helped bring aboyt some 

to V, 

of these changes as seven of eleven were implement?ed , For example, a major reason 
foryt^e quicker ac<^liiHd'tization was a thorough orientation program that included , 
visits to all^ artist/critic studios. Several felt that this feature also helped 
^make the , critiques more helpful because "you could tell where they (the artistf 
crdticsj were coming -from and interpret more easily.". Another factor in' t'he im- 
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proved critiques w^s additional visits and use of a f ixed'.schedule during ther. visits 
This way, no one got .left out while others received an hour's attention. Hovyever, 
it is always presumptious in research to claim that a report brought about. great • 
changes or helped. save^a program/" Thus", we are a bit uncomfc^rtable in suggesting 
such things . But what is 'clear is that the 'mi^ltiple perspecti-ves strat^egy ei^loyed' 
by this Office in looking- at the Studio ATts Seipester in Ne^ York City involveci 
so many people in a variety of ways .that the SAS became better-'^nderstood . "^Wheii 
decision time came.,>the various people and groups^ involved in th| deci^on ^ 
-Empires President, the Dean of ESC Center 'for Stat'ewide Pipgrams , the Advisory 
Board for the SUNY University-Wide Council on the A^, .the .iSc^al art department ; 
liaisons -- all supported continuation. ^ ° . V *^ 
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